Studies of working memory in interrupted human-computer interaction

Distraction in the workplace is increasingly more common in the information age. Several tasks and sources of information compete for a worker's limited cognitive capacities in human-computer interaction (HCI). In some situations even very brief interruptions can have detrimental effects on memory. Nevertheless, in other situations where persons are continuously interrupted, virtually no interruption costs emerge. This dissertation attempts to reveal the mental conditions and causalities differentiating the two outcomes. The explanation, building on the theory of long-term working memory (LTWM; Ericsson and Kintsch, 1995), focuses on the active, skillful aspects of human cognition that enable the storage of task information beyond the temporary and unstable storage provided by short-term working memory (STWM). Its key postulate is called a retrieval structure an abstract, hierarchical knowledge representation built into long-term memory that can be utilized to encode, update, and retrieve products of cognitive processes carried out during skilled task performance. If certain criteria of practice and task processing are met, LTWM allows for the storage of large representations for long time periods, yet these representations can be accessed with the accuracy, reliability, and speed typical of STWM. The main thesis of the dissertation is that the ability to endure interruptions depends on the efficiency in which LTWM can be recruited for maintaing information. An observational study and a field experiment provide ecological evidence for this thesis. Mobile users were found to be able to carry out heavy interleaving and sequencing of tasks while interacting, and they exhibited several intricate time-sharing strategies to orchestrate interruptions in a way sensitive to both external and internal demands. Interruptions are inevitable, because they arise as natural consequences of the top-down and bottom-up control of multitasking. In this process the function of LTWM is to keep some representations ready for reactivation and others in a more passive state to prevent interference. The psychological reality of the main thesis received confirmatory evidence in a series of laboratory experiments. They indicate that after encoding into LTWM, task representations are safeguarded from interruptions, regardless of their intensity, complexity, or pacing. However, when LTWM cannot be deployed, the problems posed by interference in long-term memory and the limited capacity of the STWM surface. A major contribution of the dissertation is the analysis of when users must resort to poorer maintenance strategies, like temporal cues and STWM-based rehearsal. First, one experiment showed that task orientations can be associated with radically different patterns of retrieval cue encodings. Thus the nature of the processing of the interface determines which features will be available as retrieval cues and which must be maintained by other means. In another study it was demonstrated that if the speed of encoding into LTWM, a skill-dependent parameter, is slower than the processing speed allowed for by the task, interruption costs emerge. Contrary to the predictions of competing theories, these costs turned out to involve intrusions in addition to omissions. Finally, it was learned that in rapid visually oriented interaction, perceptual-procedural expectations guide task resumption, and neither STWM nor LTWM are utilized due to the fact that access is too slow. These findings imply a change in thinking about the design of interfaces. Several novel principles of design are presented, basing on the idea of supporting the deployment of LTWM in the main task.

Semantic Fluency in Mild and Moderate Alzheimer's Disease

Alzheimer's disease (AD) is characterized by an impairment of the semantic memory responsible for processing meaning-related knowledge. This study was aimed at examining how Finnish-speaking healthy elderly subjects (n = 30) and mildly (n=20) and moderately (n = 20) demented AD patients utilize semantic knowledge to performa semantic fluency task, a method of studying semantic memory. In this task subjects are typically given 60 seconds to generate words belonging to the semantic category of animals. Successful task performance requires fast retrieval of subcategory exemplars in clusters (e.g., farm animals: 'cow', 'horse', 'sheep') and switching between subcategories (e.g., pets, water animals, birds, rodents). In this study, thescope of the task was extended to cover various noun and verb categories. The results indicated that, compared with normal controls, both mildly and moderately demented AD patients showed reduced word production, limited clustering and switching, narrowed semantic space, and an increase in errors, particularly perseverations. However, the size of the clusters, the proportion of clustered words, and the frequency and prototypicality of words remained relatively similar across the subject groups. Although the moderately demented patients showed a poor eroverall performance than the mildly demented patients in the individual categories, the error analysis appeared unaffected by the severity of AD. The results indicate a semantically rather coherent performance but less specific, effective, and flexible functioning of the semantic memory in mild and moderate AD patients. The findings are discussed in relation to recent theories of word production and semantic representation. Keywords: semantic fluency, clustering, switching, semantic category, nouns, verbs, Alzheimer's disease

Human brain networks of auditory attention and working memory

This thesis examines brain networks involved in auditory attention and auditory working memory using measures of task performance, brain activity, and neuroanatomical connectivity. Auditory orienting and maintenance of attention were compared with visual orienting and maintenance of attention, and top-down controlled attention was compared to bottom-up triggered attention in audition. Moreover, the effects of cognitive load on performance and brain activity were studied using an auditory working memory task. Corbetta and Shulman s (2002) model of visual attention suggests that what is known as the dorsal attention system (intraparietal sulcus/superior parietal lobule, IPS/SPL and frontal eye field, FEF) is involved in the control of top-down controlled attention, whereas what is known as the ventral attention system (temporo-parietal junction, TPJ and areas of the inferior/middle frontal gyrus, IFG/MFG) is involved in bottom-up triggered attention. The present results show that top-down controlled auditory attention also activates IPS/SPL and FEF. Furthermore, in audition, TPJ and IFG/MFG were activated not only by bottom-up triggered attention, but also by top-down controlled attention. In addition, the posterior cerebellum and thalamus were activated by top-down controlled attention shifts and the ventromedial prefrontal cortex (VMPFC) was activated by to-be-ignored, but attention-catching salient changes in auditory input streams. VMPFC may be involved in the evaluation of environmental events causing the bottom-up triggered engagement of attention. Auditory working memory activated a brain network that largely overlapped with the one activated by top-down controlled attention. The present results also provide further evidence of the role of the cerebellum in cognitive processing: During auditory working memory tasks, both activity in the posterior cerebellum (the crus I/II) and reaction speed increased when the cognitive load increased. Based on the present results and earlier theories on the role of the cerebellum in cognitive processing, the function of the posterior cerebellum in cognitive tasks may be related to the optimization of response speed.

Tarkkaavaisuuden vaikutus puheen merkityssisällön käsittelyyn: Aivojen herätevastetutkimus

Semantic processing can be studied with semantic priming. Target words that are preceded by semantically related prime words are recognized faster and more accurately than targets preceded by unrelated prime words. Semantic priming also affects the magnitude of the N400 event-related potential. The response is smaller to a target word when it is preceded by a related than an unrelated prime word. The effect is called the N400 effect. It is not yet clear, however, how attention modulates semantic priming and the N400 effect. This study investigated how the direction of attention affects the semantic processing of speech. The N400 effect was studied in experimental conditions in which the subjects attention was directed 1) away from the speech stimuli, 2) to phonological features of the speech stimuli, and 3) to semantic features of the speech stimuli. The first aim of the study was to investigate whether the N400 effect for spoken words is dependent on attention to the auditory information. The second aim was to study the differences in the N400 effect when attention is directed to the semantic or other features of speech stimuli. The results showed an N400 effect even when attention was directed away from the speech stimuli. The N400 effect was, however, stronger in conditions during which the speech stimuli were attended. The magnitude of the behavioral semantic priming and the N400 effect did not differ between the conditions during which attention was directed to the semantic or phonological features of the words. The findings indicate that the semantic processing of spoken words is not dependent on attention to auditory information. Furthermore, the results suggest that whether or not semantic processing is relevant for the task performance does not affect the semantic processing of attended spoken words.

Expert Identity in Development of Core-Task-Oriented Working Practices for Mastering Demanding Situations

The point of departure in this dissertation was the practical safety problem of unanticipated, unfamiliar events and unexpected changes in the environment, the demanding situations which the operators should take care of in the complex socio-technical systems. The aim of this thesis was to increase the understanding of demanding situations and of the resources for coping with these situations by presenting a new construct, a conceptual model called Expert Identity (ExId) as a way to open up new solutions to the problem of demanding situations and by testing the model in empirical studies on operator work. The premises of the Core-Task Analysis (CTA) framework were adopted as a starting point: core-task oriented working practices promote the system efficiency (incl. safety, productivity and well-being targets) and that should be supported. The negative effects of stress were summarised and the possible countermeasures related to the operators' personal resources such as experience, expertise, sense of control, conceptions of work and self etc. were considered. ExId was proposed as a way to bring emotional-energetic depth into the work analysis and to supplement CTA-based practical methods to discover development challenges and to contribute to the development of complex socio-technical systems. The potential of ExId to promote understanding of operator work was demonstrated in the context of the six empirical studies on operator work. Each of these studies had its own practical objectives within the corresponding quite broad focuses of the studies. The concluding research questions were: 1) Are the assumptions made in ExId on the basis of the different theories and previous studies supported by the empirical findings? 2) Does the ExId construct promote understanding of the operator work in empirical studies? 3) What are the strengths and weaknesses of the ExId construct? The layers and the assumptions of the development of expert identity appeared to gain evidence. The new conceptual model worked as a part of an analysis of different kinds of data, as a part of different methods used for different purposes, in different work contexts. The results showed that the operators had problems in taking care of the core task resulting from the discrepancy between the demands and resources (either personal or external). The changes of work, the difficulties in reaching the real content of work in the organisation and the limits of the practical means of support had complicated the problem and limited the possibilities of the development actions within the case organisations. Personal resources seemed to be sensitive to the changes, adaptation is taking place, but not deeply or quickly enough. Furthermore, the results showed several characteristics of the studied contexts that complicated the operators' possibilities to grow into or with the demands and to develop practices, expertise and expert identity matching the core task. They were: discontinuation of the work demands, discrepancy between conceptions of work held in the other parts of organisation, visions and the reality faced by the operators, emphasis on the individual efforts and situational solutions. The potential of ExId to open up new paths to solving the problem of the demanding situations and its ability to enable studies on practices in the field was considered in the discussion. The results were interpreted as promising enough to encourage the conduction of further studies on ExId. This dissertation proposes especially contribution to supporting the workers in recognising the changing demands and their possibilities for growing with them when aiming to support human performance in complex socio-technical systems, both in designing the systems and solving the existing problems.

Do wordnets also improve human performance on NLP tasks?

FinnWordNet is a wordnet for Finnish that complies with the format of the Princeton WordNet (PWN) (Fellbaum, 1998). It was built by translating the PrincetonWordNet 3.0 synsets into Finnish by human translators. It is open source and contains 117000 synsets. The Finnish translations were inserted into the PWN structure resulting in a bilingual lexical database. In natural language processing (NLP), wordnets have been used for infusing computers with semantic knowledge assuming that humans already have a sufficient amount of this knowledge. In this paper we present a case study of using wordnets as an electronic dictionary. We tested whether native Finnish speakers benefit from using a wordnet while completing English sentence completion tasks. We found that using either an English wordnet or a bilingual English Finnish wordnet significantly improves performance in the task. This should be taken into account when setting standards and comparing human and computer performance on these tasks.